Comparison of ICC and CCC for assessing agreement for data without and with replications

The intraclass correlation coefficient (ICC) has been traditionally used for assessing reliability between multiple observers for data with or without replications. Definitions of different versions of ICCs depend on the assumptions of specific ANOVA models. The parameter estimator for the ICC is usually based on the method of moments with the underlying assumed ANOVA model. This estimator is consistent only if the ANOVA model assumptions hold. Often these ANOVA assumptions are not met in practice and researchers may compute these estimates without verifying the assumptions. ICC is biased if the ANOVA assumptions are not met. We compute the expected value of the ICC estimator under a very general model to get a sense of the population parameter that the ICC estimator provides. We compare this expected value to another popular agreement index, concordance correlation coefficient (CCC), which is defined without ANOVA assumptions. The main findings are reported for data without replication and with replications for three types of ICCs defined by one-way ANOVA model, two-way ANOVA model without interaction and two-way ANOVA model with interaction. A blood pressure example is used for illustration. If the ICC is the choice of agreement index, we recommend to use ICC 3 over other ICCs as its estimate is similar to the estimate of CCC regardless whether the ANOVA assumptions are met or not.